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Description 

The present invention relates to asynchronous re- 
mote data copying or duplexing to provide data preser- 
vation in an infornnation handling system, whereby in- s 
fprmation from a primary site storage subsystem can be 
copied to a remote location. 

Data copying is one form of data preservation in an 
information handling or computer system. However, da- 
ta preservation via data copying must take many factors io 
into account. This is of special significance where it is 
anticipated that data copied and stored at a remote site 
would be the repository for any continued interaction 
with the data should the work and data of a primary site 
become unavailable. The factors of interest in copying ^5 
include the protection domain (system and/or environ- 
mental failure or device and/or media failure), data loss 
(no loss/partial loss), time where copying occurs as re- 
lated to the occurrence of other data and processes 
(point in time/real time), the degree of disruption to ap- 20 
plications executing on said computer, and whether the 
copy is application or storage system based. With re- 
gard to the last factor, application based copying in- 
volves log files, data files, and program routines while 
storage based copying involves an understanding of di- 25 
rect access storage device (DASD) addresses with no 
knowledge of data types or application use of the data. 

Real-time remote data duplexing systems require 
some means to ensure update sequence integrity as 
write updates to the secondary or remote DASD data 30 
copy. One way to accomplish this is to provide a syn- 
chronous system to control the DASD subsystems. In 
such a system, the primary DASD write operation does 
not complete until a copy of that data has been con- 
firmed at a secondary location. The problem with such 35 
synchronous systems is that they slow down the overall 
operation of the duplexing system. 

Asynchronous copy systems accomplish sequence 
integrity through a centralization and consolidation of 
data communications between primary and secondary 40 
DASD subsystems through a central communications 
system. In such systems, a system at the primary site 
can determine the sequence among different update 
write operations among all DASD subsystems at the pri- 
mary site and communicate that information to the ^5 
DASD subsystem at the remote site. The secondary 
subsystem in turn uses the sequence information from 
the primary to control the application of update data to 
the secondary DASD data copy Known asynchronous 
copy systems that utilize centralized data communica- so 
tions are described below. 

Mcllvain and Shomler, U.S. patent application 
number 08/036,017 entitled "Method and Means for 
Multi-System Remote Data Duplexing and Recovery" 
(EPA 61 7362) describes the use of a store and forward 55 
message interface at the DASD storage management 
level between a source of update copies and a remote 
site In a host to host coupling in which the difference In 



update completeness or loss of the sequence of write 
updates could be completely specified in the event of 
interruption. 

Cheffetz, et al., U.S. Patent 5,133.065 entitled 
"Backup Computer Program for Networks" issued July 
21, 1992, discloses a local area network (LAN) having 
a file server to which each local node creates and trans- 
mits a list of local files to be backed-up. Such remote 
generation reduces the traffic where a network server 
initiates the list creation and file copying activity. Argu- 
ably, art published before this reference taught centrally 
administered file selection. This resulted in compromis- 
es to local node security and overuse of the server. This 
is presumptively avoided by Cheffetz's local node gen- 
erated lists and remission of the lists to the file server. 

Beale. et al., U.S. Patent 5.155.845 entitled "Data 
Storage System for Providing Redundant Copies of Da- 
ta on Different Disk Drives", copies variable length 
records (CKD) on two or more external stores by caus- 
ing a write to be processed by the first storage controller 
and be communicated in parallel over a direct link (broad 
band path) to the second storage controller obviating the 
path length limitation between the primary and remote 
copy sites. Such a limitation is occasioned by the fact 
that CKD demand/response architecture is length limit- 
ed to in the range of 150 meters. 

Another example of an asynchronous system that 
utilizes a centralized system is disclosed in U.S patent 
application entitled "Remote Data Duplexing Asynchro- 
nous Information Packet Message, by Micka et. al. (EPA 
602822). This system discloses a system for asynchro- 
nously duplexing direct access storage device (DASD) 
data in a plurality of DASD subsystems and has the ad- 
vantage of decoupling the data duplexing operation 
from the DASD write I/O operation. This ensures the 
write does not Incur unnecessary wait states in the sub- 
system. By establishing a sequence checkpoint at which 
time a set of information packets are grouped together 
and processed as a single sequence unit, this decou- 
pling and independent operation takes place. Through 
this independence, data copying to a secondary location 
can take place without affecting the performance of the 
subsystems and also not affecting the corresponding in- 
tegrity of the data that is being updated. 

EP-A-602822 describes a system for asynchro- 
nously duplexing direct access storage device (DASD) 
data in a plurality of DASD subsystems. Advantageous- 
ly, the system decouples the data duplexing operation 
from the DASD write I/O operation, thereby ensuring 
that a write does not incur unnecessary wait states in a 
subsystem. The system uses a sequence checkpoint as 
part of this decoupling process. 

There are systems in use in which there is no cen- 
tralized communication service between the primary 
and secondary locations. Oftentimes in such system 
configurations each primary subsystem has a direct in- 
dependent link to a selected secondary subsystem. In 
such a system that Includes subsystems providing se- 



20 



25 



30 



35 



3 



EP0 674 263 B1 



4 



quence consistent asynchronous write operations can 
not be addressed utilizing known asynchronous copy 
schemes. 

Accordingly, the invention provides a method of op- 
erating an asynchronous remote data copy system in- 
cluding a primary site having a first plurality of storage 
subsystems interconnected by a first coupling means, 
and a secondary site remote from the primary site hav- 
ing a second plurality of storage subsystems intercon- 
nected by a second coupling means, each of the second 
plurality of storage subsystems being independently 
coupled to a counterpart one of the first plurality of stor- 
age subsystems, said method comprising the steps of; 

in the first plurality of subsystems; 
sending a checkpoint signal to each of the first plu- 
rality of subsystems; 

sending data updates and the checkpoint signal to 
each of the counterpart coupled second plurality of 
subsystems; and 

in the second plurality of subsystems; 
receiving and buffering the data updates and check- 
point signals; 

and coordinating the writing of the data updates 
based upon the checkpoint signals such that update 
sequence integrity is preserved at the secondary 
site. 

Such method provides independent links between 
primary and secondary subsystems, with no central 
communications system, and provides for sequence 
consistent remote copying from one set of multiple sub- 
systems at one location to a second set of multiple sub- 
systems at a second subsystem. The asynchronous 
copy system is simple, cost effective and does not sig- 
nificantly impede the overall operation of the subsys- 
tems. 

In a preferred embodiment, the step of sending a 

checkpoint signal to each of the first plurality of subsys- 
tems further comprises the step of distributing a se- 
quence signal at the primary site, said checkpoint signal 
including information indicative of the value of the se- 
. quence signal. Such a sequence signal is used to en- 
sure synchronisation between the various subsystems. 
However, if the network can ensure quick and reliable 
distribution of the checkpoint messages, then In some 
situations it is possible to dispense with the sequence 
signal. 

It is further preferred that said step of sending data 
updates and a checkpoint signal to each of the counter- 
part coupled second plurality of subsystems comprises 
the steps of: generating asynchronously a sequence of 
data updates at each of said first plurality of subsystems; 
ordering each of said sequences in accordance with the 
distributed sequence signal and checkpoint signal; and 
asynchronously communicating sequences of updates 
from each of the first plurality of subsystems to a coun- 
terpart subsystem. 



In a preferred embodiment, the method furthercom- 
prises the initial steps at the primary site of: activating 
each of the first plurality of subsystems to communicate 
with the other subsystems in said first plurality of sub- 

5 systems; activating each of the first plurality of subsys- 
tems to communicate with its counterpart coupled sub- 
system in the second plurality of subsystems; building 
at least one configuration table in each of the first plu- 
rality of subsystems such that each of the first plurality 

10 of subsystems can identify all of the other subsystems 
in said first plurality of subsystems; and synchronizing 
the first plurality of subsystems; 

and the method further comprises the initial steps 
IS at the secondary site of: receiving copy active mes- 
sages in the second plurality of subsystems; build- 
ing copy active tables in the second plurality of sub- 
systems; and synchronizing copy operations of the 
second plurality of subsystems; 
20 and wherein the coordinating step further compris- 
es the step of performing a rendezvous for all 
checkpoint signals in the second plurality of subsys- 
tems. 

25 Also in a preferred embodiment, the method further 
comprises the Initial steps of: 

at the primary site responsive to initiation of a start 
copy operation, ascertaining at each of the subsys- 

30 tems in said first plurality of subsystems, the subset 
of the plurality of subsystems forming a copyset 
group, designating one of the plurality of subsys- 
tems as a clocking and checkpoint signal source, 
each checkpoint signal including a sequence clock 

3S value and an increased sequence number; 

at the secondary site responsive to initiation of a 
start copy operation, ascertaining at each of the 
subsystems in said second plurality of subsystems, 
the subset of the plurality of subsystems forming a 

40 copyset group, and designating one of the plurality 
of subsystems as checkpoint signal rendez-vous 
point for synchronising the second plurality of sub- 
systems; 

and wherein the step of sending a checkpoint signal 
45 to each of the first plurality of subsystems compris- 
es the steps of periodically generating clocking sig- 
nals and checkpoint signals by said designated 
subsystem at the primary site and broadcasting 
said clocking and checkpoint signals as they occur 
so to other subsystems forming the copyset group at 
the primary site including itself; 
and wherein the step of sending data updates and 
the checkpoint signal to each of the counterpart 
coupled second plurality of subsystems comprises 
55 for each subsystem in the copyset group, asynchro- 
nously forming a local sequence of updated 
records, embedding said clocking and checkpoint 
signals into the sequence to form a time discrimi- 
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nated total ordering of updated records, and send- 
ing at least a portion of the sequence to the coun- 
terpart subsystem at the secondary site; 
and wherein the step of coordinating the writing of 
data updates and checkpoint signals at the second- 5 
ary site comprises the steps of sending the check- 
point signal to the designated subsystem at the sec- 
ondary site by each subsystem in the copyset 
group, and writing the sequences or portions there- 
of buffered in said subsystems only upon a signal 
from said designated subsystem indicative of its re- 
ceipt of all the checkpoint signals sent to it. 

Each checkpoint signal signifies that all data up- 
dates with a clock signal with a sequence number that 
is less than the checkpoint signal sequence clock value 
have been transmitted to the counterpart secondary 
subsystem. 

The invention further provides an asynchronous re- 
mote data copy system including a primary site having 
a first plurality of storage subsystems interconnected by 
a first coupling means, and a secondary site remote 
from the primary site having a second plurality of storage 
subsystems interconnected by a second coupling 
means, each of the second plurality of storage subsys- 
tems being independently coupled to a counterpart one 
of the first plurality of storage subsystems, 

the first plurality of subsystems including means for 
sending a checkpoint signal to each of the first plu- 
rality of subsystems; and means for sending data 
updates and the checkpoint signal to each of the 
counterpart coupled second plurality of subsys- 
tems; and 

the second plurality of subsystems including means 
for receiving and buffering the data updates and 
checkpoint signals; and means for coordinating the 
writing of the data updates based upon the check- 
point signals such that update sequence integrity is 
preserved at the secondary site. 

Preferably, the means for sending data updates and 
the checkpoint signal to each of the counterpart coupled 
second plurality of subsystems comprises means, re- 
sponsive to the means for sending a checkpoint signal 
to each of the first plurality of subsystems, for inserting 
the checkpoint signal into a data update sequence from 
each of said first plurality of subsystems to its counter- 
part subsystem. 

Thus the above method and means may be used 
for the remote copying of data at a secondary DASD 
subsystems. Typically in such an implementation, the 
secondary DASD subsystems are peer coupled to coun- 
terpart ones of a primary host attached DASD subsys- 
tems. A start copy operation is initiated at the host by a 
message broadcast to all primary DASD subsystems. 
Each primary builds a configuration table using a local 
area network or other suitable means as a set associa- 



tive device for table building. This table identifies all the 
primary subsystem participants. Also, each primary syn- 
chronizes its local clock or other time reference to that 
of a designated primary subsystem, the designated pri- 
mary being operative as a clocking and checkpoint mes- 
sage source. The designated primary periodically sends 
checkpoint messages having a sequence time value 
and an increasing checkpoint sequence number, to 
each primary subsystem. These are logically inserted 
into its local copy write record sequence. At the second- 
ary host attached DASD subsystems, one of the sec- 
ondary subsystems is designated as a local synch 
source. Each secondary subsystem builds a configura- 
tion table of copy active subsystems, and couples to the 
counterpart primary subsystem. Next, each secondary 
subsystem asynchronously receives and locally buffers 
a copy sequence from its primary counterpart. Each re- 
ceived sequence includes checkpoint messages em- 
bedded therein at the primary after all time stamped 
write updated records have been sent. Responsive to 
receipt of a checkpoint message, the secondary sub- 
system will remit it to the local synch source. A rendez- 
vous is executed by this source over all the checkpoint 
messages. Thus, each secondary subsystem writes 
from buffer to DASD upon receipt of the rendezvous 
completion from the synch source. If one or more of the 
primary DASD subsystems become unavailable, then 
their counterparts at the secondary site resign from the 
copygroup. Such occurs only after completion of any up- 
dates in progress. 

This approach allows for distributed non-central- 
system operated control of a sequence-consistent real- 
time asynchronous data copy from a set of DASD sub- 
systems at a primary location to a set of DASD subsys- 
tems at a secondary location, with primary subsystems 
separately and independently connected to secondary 
subsystems. This enables update sequence integrity of 
a data copy at a plurality of subsystems remote from a 
source of asynchronously independently generated se- 
quence of write operations, there being a first plurality 
of subsyistems at a primary site, the first plurality of sub- 
systems being interconnected by a first coupling means, 
and a second plurality of subsystems at a site remote 
from the primary site, the second plurality of subsystems 
being interconnected by a second coupling means, 
each of the second plurality of subsystems being inde- 
pendently coupled to one of the first plurality of subsys- 
tems. 

Thus in a system for remote copying of data at a 
secondary site having a coupling plurality of storage de- 
vice (DAS) subsystems that are interconnected by a first 
couplingmeans, said secondary DASD subsystems be- 
ing coupled to counterpart ones of a plurality of DASD 
subsystems at a remote primary site, said secondary 
DASD subsystems being interconnected by a second 
coupling means, said system including means at the pri- 
mary site for initiating a start copy operation, a method 
Is provided comprising the steps of: 
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(a) at the primary and secondary sites, forming m 
copyset groups of DASD subsystems respectively; 

(b) at the primary site: 

(1 ) causing each of the m subsystems to asyn- 
chronously generate a sequence of updated 
write records; 

(2) ordering each of said sequences by embed* 

ding common clock values and a periodic 
checkpoint message with a common clock val- 
ue and increasing a sequence number in each 
of said sequences, 

(3) coupling counterpart DASD subsystems be- 
tween the sites and asynchronously communi- 
cating sequences from each of the primary site 
subsystems into a buffered portion of a coun- 
terpart secondary site subsystem; and 

(c) at the secondary site, writing the buffered se- 
quences to DASD in the counterpart subsystems as 
a function of each checkpoint message. 

An embodiment of the invention will now be de- 
scribed in detail by way of example only, with reference 

to the following drawings; 

Figure 1 is a conventional remote data copy system 
configuration; 

Figure 2 is remote dual copy system configured In 
accordance with the present invention; 
Figure 3 is a flow chart showing the general opera- 
tion of the remote dual copying of Figure 2; and 
Figure 3A-3D are more detailed flow charts showing 
the operation of the remote dual copying system of 
Figure 2. 

To perform asynchronous remote copying, (1) the 
sequence of data updates must be determinable at a 
local site; (2) that sequence must be communicable by 
the local site to a remote site; and (3) the remote site 
must be able to use the sequence to control the updating 
at the remote site. 

As has been before mentioned, prior art asynchro- 
nous copy systems which include multiple subsystem 
require a central system for providing the appropriate 
sequences of data. However, system configurations 
may be found in which it is not desired to pass update 
data between the primary and secondary locations 
through a centralized communication service. Rather in 
those configurations it is desired to directly connect 
DASD subsystems at the primary and secondary sites 
via independent subsystem-to-subsystem communica- 
tion links. Such a system 10 is shown In Figure 1. The 
system 1 0 includes a host 11 which provides data to pri- 
mary subsystems 12'. As is shown each of the links be- 
tween a primary DASD subsystem 1 2' and its peer cou- 
pled secondary DASD subsystem 14' is independent. 
As a result, this type of system would inherently be in- 



capable of write operations that are sequentially con- 
sistent. 

To more specifically describe the problem, consider 
as an example a sequence of three writes from a con- 
s ventional database management system (DBMS) as it 
is about to commit a transaction. The example is repre- 
sentative of an information management sen/ice (IMS) 
system: 

1. The DBMS writes to its log data set; the record 
written contains old data base (DB) data, new DB 
data (that is being changed by this transaction), and 

' a record of its intent to commit (finalize) this trans- 
action. 

2. The DBMS waits for a DASD I/O operation to re- 
port that it has completed, then it updates its data 
base data sets, which are different volumes that are 
on a different DASD subsystems. This writing of the 
new DB data ovenvrites and thus destroys the old 
DB records. 

3. The DBMS waits for the second DASD I/O oper- 
ation to be posted complete, then it writes a commit 
record to its log data set on the first volume. This 
commit record 'guarantees' to future IMS recovery 
processes that the data base data set (DASD vol- 
umes) have been updated. 

Now consider operation of an asynchronous remote 
copy system using a multiple subsystem configuration 
such as illustrated in Figure 1. In this embodiment, the 
DBMS log data set is on a volume configured to the top- 
most DASD subsystem pair (DASD 1 and DASD 1 ') and 
the data base volumes are on the DASD subsystem pair 
shown second from the top, (DASD 2 and DASD 2'). In 
this example, the primary subsystem 1 is lightly loaded, 
thus it processes its queued work with no delay, while 
the primary subsystem 2 is heavily loaded and is expe- 
riencing some delay in processing its queued work. 
Queued work includes the fon/varding of updated data 
to its remote copy peer subsystem. 

The following sequence would describe the opera- 
tion in the present example: 

1 . Write I/O (1 ) is completed from application to sub- 
system 1 . 

2. Write I/O (2) Is completed from application to sub- 
system 2. 

3. Primary Subsystem 1 sends data for I/O (1) to 
secondary subsystem 1, which applies it to its 
cache memory copy of the DASD volume. 

4. Write I/O (3) is completed from application sys- 
tem to subsystem 1 . 

5. Primary Subsystem 1 sends data for I/O (3) to 
secondary subsystem 1, which applies it to its 
cache memory copy of the DASD volume. 

6. Primary Subsystem 2 sends data for I/O (2) to 
secondary subsystem 2, which applies to its cache 
memory copy of the DASD volume. 
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If a primary site failure occurs after step 5 and be- 
fore step 6 there would be corrupted data copied into 
the system. Since the failure rendered primary subsys- 
tems inaccessible, the data from I/O (2) will not be at 
the secondary site. 

The essence of data sequence consistency In a re- 
mote copy sen^ice is to ensure at such a time as the 
remote DASD must be used for recovery, that the new 
data from second operation I/O above will only be seen 
if the data from the first I/O is also present, and that the 
data from the third I/O will only be present if the second 
I/O is also present. Consider the data integrity loss if the 
third I/O were present but if the second I/O was not. The 
DBMS log would tell recovery processes that the data 
base would receive valid data. This would either result 
in a DBMS failure or business application error. 

The sequence of I/O operations at the primary sub- 
systems is DASD 1 , DASD 2, then back to DASD 1 ; but 
because of the load on DASD subsystem 2, it is delayed 
In sending its I/O (2) such that it has not arrived by the 
time I/O (3) from DASD subsystem 1 was received by 
secondary DASD subsystem 1 With no control of trans- 
mission subsystem data to copy volumes, subsystem 1 
would update its copy volume with both 1 and 3. If at 
that time a disaster befell that primary system and op- 
erations were directed to resume at the secondary site 
(such contingency being the reason for having a real- 
time remote DASD copy), the recovering DBMS would 
find a log record that said that data base records were 
written (in I/O 2) while the data for I/O 2 would not be on 
the DASD of secondary subsystem 2. 

With such independent links, determination of se- 
quence of writes among independent primary DASD 
subsystems, communication of that information to the 
set of subsystems at the secondary location, and control 
of the sequence of updates among the independent 
DASD subsystems at the secondary has been a prob- 
lem heretofore. However, as explained in detail below, 
a system and method are provided for sequence iden- 
tification, communication, and update control that permit 
sequence-consistent remote DASD data copy from mul- 
tiple independent DASD subsystems at one location to 
a set of secondary DASD subsystems, each subsystem 
being independently connected to peer subsystems at 
the first location. 

Figure 2 is a remote dual copy system 20 including 
a host 11 which provides data to primary subsystems 
12. The system 20 includes a first group of DASD sub- 
systems 12 which are located at a primary site and a 
second group of DASD subsystems 14 which are locat- 
ed at a site that is remote from the primary site. The 
system 20 includes couplers 16 and 18 which provide 
for interconnections of the DASD subsystems 12 and 
14, respectively. 

Referring now to Figure 2, system 20 for achieving 
update sequence integrity without a centralized commu- 
nication system is based on the presence of the follow- 
ing configurations. 



1 . Multiple primary location DASD subsystems each 
interconnected via one or more communication 
links to a peer DASD subsystem at the secondary 
location. 

S 2. A coupling connection 1 6 of all subsystems at the 
primary site, and a similar coupling connection 18 
of all subsystems at the secondary site. While not 
shown, one of ordinary skill in the art will readily rec- 
ognize multiple physical connections may be incor- 
porated for connection redundancy. The term "cou- 
pling" is used for convenience; any suitable physical 
network interconnection means may be used such 
as a local area network (LAN) or the like. 
3. Each subsystem has a 'clock' or similar synchro- 
nizing signal process that can be synchronized with 
a value from another subsystem, communicated via 
the coupling 16. 

Three steps are utilized to achieve copy update se- 
quence integrity: (1 ) determination of sequence of write 
operations among all DASD subsystems at the primary; 
(2) communication of that sequence information to the 
secondary; and (3) use of that Information by the sec- 
ondary DASD subsystems to control the sequence of 
update writes across all secondary DASD. These three 
steps are described in more detail hereinbelow for a 
copy system using LAN interconnections at the second- 
ary and primary subsystems. 

1 . Use of the LAN interconnection among the pri- 
mary DASDs to distribute a sequencing signal, such 
as a time clock, to all subsystems is used to asso- 
ciate a sequenceAime value with each DASD write 
of data to be copied to a secondary DASD subsys- 
tem (and the sending of that value along with update 
data to the secondary); 

2. Propagation of periodic synch ronizing-time-de- 
nominated checkpoint signals among the primary 
DASD subsystems that in turn are communicated 
by each primary subsystem to its peer-connected 
secondary subsystem(s); and 

3. Use of the LAN interconnection among second- 
ary DASD subsystems to coordinate their DASD up- 
date writing of copy data received from primary sub- 
systems. 

Referring now to Figure 3, what is shown is a flow 
chart of the general operation 1 00 of such a system that 
is located within the primary subsystem. First, a start 
copy operation is sent to all the primary subsystems to 
activate communication between primary subsystems, 
via step 102. Then individual subsystems provide the 
appropriate sequence information to all the copy active 
primary subsystems, via step 104. Thereafter the prima- 
ry subsystems communicate the sequence information 
to the peer coupled secondary subsystems, via step 
106. Finally, that sequence information Is utilized to con- 
trol secondary subsystem updates via step 108. 
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These steps are described in detail below: 

Start copy operation (step 102) 

Remote copy operations for each DASD subsystem s 
must be started by some instruction to that subsystem. 
The instruction may come via command from a host sys- 
tem (the same system as may be executing applications 
that write to DASD), or it may be provided via subsys- 
tem-local interface such as from a subsystem operator 
console or similar means. 

Description of DASD write sequence at the primary 
(step 104) 

Refer now to Figure 3A and the following discus- 
sion. Irrespective of whether the command comes from 
the host system or from subsystem local interface, the 
start copy instruction identifies the DASD to be copied 
and causes the subsystem to activate communications 
with secondary subsystem(s) to which it is connected. 

The start copy operation also activates communi- 
cation from that DASD subsystem to other primary 
DASD subsystems (step 202). The LAN connection ad- 
dresses of other primary DASD subsystems may be 
configured to each subsystem or it may be incorporated 
in information contained in the start copy instruction to 
the subsystem. When copy is started at a subsystem, 
this subsystem sends a subsystem copy active mes- 
sage to all other primary subsystems it has addresses 
for via step 204. All primary subsystems build and main- 
tain configuration tables such that each subsystem 
knows the identity of all primary subsystems participat- 
ing in the copy process via step 206. Each subsystem 
receiving an copy active message from another subsys- 
tem marks that subsystem as a copy active in its con- 
figuration tables. 

As a part of exchanging copy active messages with 
the other primary systems, the subsystems synchronize 
their sequence clock processes and select one subsys- 
tem to be a master source for a timing value synchroni- 
zation signal via step 208. This is a clock synchroniza- 
tion process, not described here since such processes 
are well known in the art. Note that clock synchroniza- 
tion must be able to maintain clock drift such that max- 
imum drift is substantially less than the time for a sub- 
system to receive a write command from a system, per- 
form a write to cache, signal end of I/O operation, and 
for the host, to process the I/O completion status and 
start a new DASD I/O write operation. 

As write operations are performed by each subsys- 
tem, in a preferred embodiment, the subsystem includes 
the then current time sequence signal value with other 
control information that is sent with the DASD data to Its 
connected secondary subsystem. The primary system 
DASD write I/O operation continues without delay, ex- 
tended only by the time necessary to construct control 
information for the data to be sent to the secondary. 



DASD and control data are buffered in the secondary 
subsystem on receipt. Update of secondary DASD copy 
is deferred until released by secondary sequence con- 
trol (described below). 

Communication of sequence information to secondary 
DASD subsystems (step 106) 

Refer now to Figure 3B and the following discus- 
sion. The subsystem that is providing the time synchro- 
nizing signal source will periodically send a checkpoint 
message to all primary copy-active subsystems via step 
302. This may be included with the time sync signal or 
sent separately as appropriate for the local interconnec- 
tion protocol used. The checkpoint message includes a 
sequence time value and a checkpoint sequence 
number that is incremented by one for each checkpoint 
message. Each subsystem on receiving the checkpoint 
communication will logically insert it In its transmission 
stream to the secondary subsystem(s) to which it is con- 
nected, sending it to secondary subsystem(s) only after 
all DASD and control information with an earlier se- 
quence time signal value have been sent via step 304. 

Use of sequence information by secondary DASD 
subsystems to control secondary DASD updates (step 



Refer now to Figures 3C and 3D and the following 
discussion. Copy operations in secondary systems are 
activated by copy active messages from their connected 
primary subsystems via step 402. (An enabling startup 
instruction from a system or local console at the second- 
ary may also be required. That aspect of copy operation 
and control is performed in a conventional manner) 
Secondary subsystems build and maintain copy active 
tables such that each subsystem has data relating to the 
identity of all secondary subsystems participating in the 
copy process, via step 404. A synchronizing control 
master is selected from among the secondary subsys- 
tems, using the local interconnect (eg a LAN) similar to 
the manner in which a primary synchronizing signal 
source was selected, via step 406. Such distributed con- 
trol schemes are well known and need not be described 
here. 

Secondary subsystems receive and buffer DASD 
data and associated control information from their con- 
nected primary subsystems. This received data and 
control information is logically grouped and sequenced 
by the primary synchronizing signal value. For maxi- 
mum protection, received data and control information 
buffering should be in a non-volatile storage medium. 

At some point, each subsystem will receive a check- 
point control message via step 408 that signifies that all 
DASD update data and control with a primary synchro- 
nizing signal value equal to or less than the checkpoint 
time sync value has been sent to that secondary sub- 
system. A checkpoint message, when received, is sent 
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by the receiving subsystem to the secondary master 
subsystem, which performs a rendezvous for that 
checkpoint message from all copy-active secondary 
subsystems, including itself, via step 410. 

When the rendezvous is complete via step 41 2 for s 
a given checkpoint value, the secondary master subsys- 
tem sends a message to all secondary copy-active sub- 
systems to release update data up to the primary se- 
quence time value of the checkpoint. Each subsystem, 
then marks itself in an internal update in progress state io 
and applies the updates to its secondary D ASD copy via 
step 41 4. (Note: The actual process of applying the buff- 
ered DASD copy data may require only adjusting cache 
directory entries.) 

The update in progress state must be maintained 
through subsystem resets and power off along with buff- 
ered data and control info from the primary (and other . 
copy service state information). It is used as a must- 
complete type operation control that precludes any host 
system access to secondary DASD when that state is 20 
active. This ensures that an application takeover at the 
secondary cannot see partial updated DASD data. That 
is, it can not access a sequence-inconsistent DASD 
copy. Update for the checkpoint interval must be com- 
plete before a user can access the copy DASD subsys- 2S 
tems records. 

When the updating of DASD copy data has com- 
pleted, the subsystem sends an update complete for 
checkpoint message (identifying the specific check- 
point) for that checkpoint to the secondary master, via 30 
step 416. When update complete signal for checkpoint 
messages have been received at the master from all 
subsystems including the master, the master then sends 
a reset update in progress state message to all second- 
ary subsystems to allow secondary copy data to again 35 
be accessible to attached systems, via step 41 8. 

In a variation to the preferred embodiment, if the 
coupling means among all primary subsystems can re- 
liably propagate every checkpoint message to all sub- 
systems in substantially less time than the time for an 1/ 40 
O operation cycle then the preceding processes could 
function without a clock and clock sync. The arrival time 
of a checkpoint message at each primary subsystem 
would be precise enough to define update sequence. 
All updates within a checkpoint would be considered to 45 
have occurred at the same time. 

The steps of subsystem operation for the three i/Os 
described previously, using the approach described 
above is discussed herein below. 

so 

1 . DASD 1 , 2 and 3, and 4 exchange 'copy active' 
messages. Primary subsystem 3 has become the 
master source for timing value sync signal. Second- 
ary system 4 has become the secondary master for 
rendezvous of checkpoint messages. ss 

2. Write I/O (1 ) is completed from application to sub- 
system 1 at time 'a'. 

3. Write I/O (2) is completed from application to sub- 



system 2 at time 'b'. 

4. Primary Subsystem 1 sends data for I/O (1) to 
secondary subsystem 1 along with its associated 
time value 'a'. Secondary subsystem 1 buffers the 
data but does not apply it to its cache copy for the 
DASD volume. 

5. Subsystem 3 sends a checkpoint message con- 
taining checkpoint sequence number 'n' and time 
value 'b'. 

6. Write I/O (3) is completed from application sys- 
tem to subsystem 1 at time 'c'. 

7. Primary Subsystem 1 sends data for I/O (3) to 
secondary subsystem 1 along with its associated 
time value 'c'. Secondary subsystem 1 buffers the 
data but does not apply it to its cache copy for the 
DASD volume. 

8. Primary subsystem 1 receives and processes the 
checkpoint message sent by subsystem 3 in step 5. 

9. Primary subsystem 1 sends checkpoint message 
to secondary subsystem 1 , which forwards it to sec- 
ondary subsystem 4. 

10. Primary Subsystem 2 sends data for I/O (2) to 
secondary subsystem 2 along with its associated 
time value 'b'. Secondary subsystem 2 buffers the 
data but does not apply it to its cache copy for the 
DASD volume. 

11. Primary subsystem 2 receives and processes 
the checkpoint message sent by subsystem 3 in 

step 5. 

12. Primary subsystem 2 sends checkpoint mes- 
sage to secondary subsystem 2, which fonwards it 
on to secondary subsystem 4. 

1 3. At some point between steps 5 and 1 3, subsys- 
tems 3 and 4 have sent the checkpoint message 'n' 
to their secondary subsystems. Secondary subsys- 
tem 3 has fonwarded it to secondary subsystem 4. 

14. Secondary subsystem 4 sends a 'release' mes- 
sage to secondary subsystems 1 , 2 and 3. 

15. Secondary subsystem 3 having no update im- 
mediately returns an "update complete" message 
to secondary subsystem 4. 

16.. Secondary subsystem 1 enters 'update in 
progress' state, then applies update (1 ). It does not 
apply update (3) since its sync time value 'c' is great- 
er than checkpoint time value 'b'. It then sends an 
update complete message to secondary subsystem 
4. 

17. Secondary subsystem 2 enters update state 
and applies update (2), and sends an update com- 
plete message to secondary subsystem 4. 

1 8. Secondary subsystem 4 having received update 
complete messages from all secondary subsys- 
tems, sends a 'reset update in progress state' mes- 
sage to secondary subsystems 1 , 2 and 3. 

Now if a primary site failure happens at any point in 
the above sequence the secondary DASD will either 
show none of the updates, or will show updates (1) and 
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(2). Update (3) will not be 'applied' to secondary subsys- 
tem 1 's DASD and cache until the next checkpoint se- 
quence. 

Accordingly through the present system a se- 
quence consistent real-time asynchronous copy system 
is provided that does not require the use of central com- 
munications service. In so doing, a system is provided 
that requires minimal modification, while utilizing exist- 
ing capabilities within the DASD subsystems to provide 
for sequence modifications. 



Claims 

1 . A method of operating an asynchronous remote da- 
ta copy system (20) including a primary site having 
a first plurality of storage subsystems (1 2) intercon- 
nected by a first coupling means (16), and a sec- 
ondary site remote from the primary site having a 
second plurality of storage subsystems (14) inter- 
connected by a second coupling means, each of the 

. second plurality of storage subsystems being inde- 
pendently coupled to a counterpart one of the first 
plurality of storage subsystems, said method com- 
prising the steps of; 

in the first plurality of subsystems; 
sending (302) a checkpoint signal to each of the 
first plurality of subsystems; 
sending (304) data updates and the checkpoint 
signal to each of the counterpart coupled sec- 
ond plurality of subsystems; and in the second 
plurality of subsystems; 
receiving (408) and buffering the data updates 
and checkpoint signals; 
and coordinating the writing (414) of the data 
updates based upon the checkpoint signals 
such that update sequence integrity is pre- 
sented at the secondary site. 

2. The method of claim 1 , wherein the step of sending 
a checkpoint signal to each of the first plurality of 
subsystems further comprises the step of distribut- 
ing a sequence signal at the primary site, said 
checkpoint signal including information indicative of 
the value of the sequence sjgnal. 

3. The method of claim 2, wherein said step of sending 

data updates and a checkpoint signal to each of the 
counterpart coupled second plurality of subsystems 
comprises the steps of: 

generating asynchronously a sequence of data 
updates at each of said first plurality of subsys- 
tems; 

ordering (108) each of said sequences In ac- 
cordance with the distributed sequence signal 
and checkpoint signal; and 



asynchronously communicating sequences of 
updates from each of the first plurality of sub- 
systems to a counterpart subsystem. 

5 4. The method of any preceding claim further compris- 
ing the initial steps at the primary site of : 

activating (202) each of the first plurality of sub- 
systems to communicate with the other subsys- 
10 tems in said first plurality of subsystems; 

activating each of the first plurality of subsys- 
tems to communicate with its counterpart cou- 
pled subsystem in the second plurality of sub- 
systems; 

15 building (206) at least one configuration table 

in each of the first plurality of subsystems such 
that each of the first plurality of subsystems can 
identify all of the other subsystems in said first 
plurality of subsystems; and 

20 synchronizing (208) the first plurality of subsys- 

tems. 

5. The method of any preceding claim, further com- 
prising the Initial steps at the secondary site of: 

25 

receiving (402) copy active messages in the 
second plurality of subsystems; 
building (404) copy active tables in the second 
plurality of subsystems; and 

30 synchronizing (406) copy operations of the sec- 

ond plurality of subsystems; 
' and wherein the coordinating step further com- 
prises the step of performing a rendezvous 
(410) for all checkpoint signals in the second 

35 plurality of subsystems. 

6. The method of claim 1 , further comprising the initial 
steps of: 

40 at the primary site responsive to initiation of a 

start copy operation, ascertaining at each of the 
subsystems in said first plurality of subsystems, 
the subset of the plurality of subsystems form- 
ing a copyset group, and designating one of the 

45 plurality of subsystems as a clocking and 

checkpoint signal source, each checkpoint sig- 
nal including a sequence clock value and an in- 
creased sequence number; 
at the secondary site responsive to initiation of 

50 a start copy operation, ascertaining at each of 

the subsystem in said second plurality of sub- 
systems, the subset of the plurality of subsys- 
tems forming a copyset group, and designating 
one of the plurality of subsystems as check- 

55 point signal rendez-vous point for synchronis- 

ing the second plurality of subsystems; 
and wherein the step of sending a checkpoint 
signal to each of the first plurality of subsystems 
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comprises the steps of periodically generating 
clocking signals and checkpoint signals by said 
designated subsystenn at the primary site and 
broadcasting said clocking and checkpoint sig- 
nals as they occur to other subsystems forming 5 
the copyset group at the primary site including 
itself; 

and wherein the step of sending data updates 
and the checkpoint signal to each of the coun- 
terpart coupled second plurality of subsystems io 
comprises for each subsystem in the copyset 
group, asynchronously forming a local se- 
quence of updated records, embedding said 
clocking and checkpoint signals Into the se- 
quence to form a time discriminated total order- is 
ing of updated records, and sending at least a 
portion of the sequence to the counterpart sub- 
system at the secondary site; 
and wherein the step of coordinating the writing 
of data updates and checkpoint signals at the 20 
secondary site comprises the steps of sending 
(410) the checkpoint signal to the designated 
subsystem at the secondary site by each sub- 
system in the copyset group, and writing the se- 
quences or portions thereof buffered in said 25 
subsystems only upon a signal from said des- 
ignated subsystem indicative of Its receipt of all 
the checkpoint signals sent to It (41 2). 

7. The method according to claim 6. wherein each 30 
checkpoint signal signifies that all data updates with 

a clock signal with a sequence number that is less 
than the checkpoint signal sequence clock value 
have been transmitted to the counterpart secondary 
subsystem. 35 

8. An asynchronous remote data copy system (20) in- 
cluding a primary site having a first plurality of stor- 
age subsystems (12) Interconnected by a first cou- 
pling means (16), and a secondary site remote from 40 
the primary site having a second plurality of storage 
subsystems (14) interconnected by a second cou- 
pling means, each of the second plurality of storage 
subsystems being independently coupled to a 
counterpart one of the first plurality of storage sub- 45 
systems, 

the first plurality of subsystems including 
means (12) for sending a checkpoint signal to 
each of the first plurality of subsystems; and so 
means (12) for sending data updates and the 
checkpoint signal to each of the counterpart 
coupled second plurality of subsystems; and 
the second plurality of subsystems Including 
means (14) for receiving and buffering the data 55 
updates and checkpoint signals; and means 
(14) for coordinating the writing of the data up- 
dates based upon the checkpoint signals such 



that update sequence Integrity Is presen/ed at 
the secondary site. 

9. The system of claim 8, wherein the means for send- 
ing a checkpoint signal to each of the first plurality 
of subsystems further Includes means for distribut- 
ing a sequence signal at the primary site, said 
checkpoint signal including information indicative of 
the value of the sequence signal. 

10. The system of claim 8 or 9 in which the means for 
sending data updates and the checkpoint signal to 
each of the counterpart coupled second plurality of 
subsystems comprises means, responsive to the 
means for sending a checkpoint signal to each of 
the first plurality of subsystems, for Inserting the 
checkpoint signal into a data update sequence from 
each of said first plurality of subsystems to its coun- 
terpart subsystem. 

Patentanspruche 

1. Verfahren zum Betreiben eines asynchronen Da- 
tenfernkopiersystems (20), das eine Primarstatlon 
enthalt, die eIne erste VIelzahl von Spelcher-Sub- 
systemen (1 2) hat. die durch ein erstes Kopplungs- 
mittel (16) miteinander verbunden sind, und eine 
von der Primarstation entfernte Sekundarstatlon. 
die eine zwelte Vielzahl von Speicher-Subsyste- 
men (14) hat, die durch ein zweltes Kopplungsmittel 
miteinander verbunden sind, wobei jedes der zwei- 
ten VIelzahl der Speicher-Subsysteme unabhangig 
von den anderen mit einem Gegenstuck der ersten 
Vielzahl der Speicher-Subsysteme verbunden ist, 
wobel das Verfahren die folgenden Schritte umfaBt: 

in der ersten VIelzahl der Subsysteme 

Senden (302) eines Prufpunktsignals an jedes 
der ersten Vielzahl der Subsysteme; 

Senden (304) von Datenaktualisierungen und 
des Prufpunktsignals an jedes der mit eInem 
Gegenstuck verbundenen zweiten Vielzahl der 
Subsysteme; und 

in der zwielten Vielzahl der Subsysteme 

Empfangen (408) und Puffem der Datenaktua- 
lisierungen und der Prufpunktsignale; 

und Koordinleren des Schrelbens (41 4) der Da- 
tenaktualisierungen auf der Grundlage der 
Prufpunktsignale, so daB die Integritat der Ak- 
tualisierungsfolgen an der Sekundarstatlon ge- 
wahrt bleibt. 
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2. Verfahren nach Anspruch 1, wobei der Schritt des 
Sendens eines Prufpunktsignals an jedes der er- 
sten Vielzahl der Subsysteme des weiteren den 
Schritt der Verteilung eines Folgesignals an der Pri- 
marstation umfaf3t, wobei das Prufpunktsignal In- s 
formationeh enthalt, die den Wert des Folgesignals 
anzeigen. 

3. Verfahren nach Anspruch 2, wobei der Schritt des 
Sendens von Datenaktualisierungen und eines io 
Prufpunktsignals an jedes der nnit einem Gegen- 
stuck verbundenen zweiten Vielzahl der Subsyste- 
me die folgenden Schritte umfalBt: 

Asynchrones Erzeugen einer Folge von Daten- 
aktualisierungen an jedem der ersten Vielzahl 
der Subsysteme; 

Ordnen (108) einer jeden der Folgen entspre- 
chend dem verteilten Folgesignal und dem 
Prufpunktsignal; und 

Asynchrones Ubertragen von Aktualisierungs- 
folgen von jedem der ersten Vielzahl der Sub- 
systeme an ein Subsystem-Gegenstuck. 

4. Verfahren nach einem vorhergehenden Anspruch, 
das an der Primarstation des weiteren die folgen- 
den ersten Schritte umfaBt: 



5. Verfahren nach einem vorhergehenden Anspruch, 
das an der Sekundarstation des weiteren die fol- 
genden ersten Schritte umfa3t: 

Empfangen (402) von Koplen/organg-Aktiv- 
Nachrichten in der zweiten Vielzahl der Subsy- 
steme; 



Erstellen (404) von Kopiervorgang-Aktiv-Tabei- 
len in der zweiten Vielzahl der Subsysteme; 
und 

Synchronisieren (406) von Kopieroperationen 
der zweiten Vielzahl der Subsysteme; 

und wobei der Koordinlerungsschritt des weite- 
ren den Schritt der Durchfuhrung eines Tref- 
fens (41 0) aller Prufpunktsignale in der zweiten 
Vielzahl der Subsysteme umfaBt. 

6. Verfahren nach Anspruch 1, das des weiteren die 
folgenden ersten Schritte umfa3t: 



an der Sekundarstation als Antwort auf die Ein- 
leitung einer Startkopieroperation an jedem der 
Subsysteme in der zweiten Vielzahl der Subsy- 
steme Feststellen der Untergruppe der Vielzahl 
der Subsysteme, die eine Kopiersatzgruppe 
bildet, und Bestimmen eines der Vielzahl der 
Subsysteme zu einem PrOfpunktsignal-Treff- 
punkt, um die zweite Vielzahl der Subsysteme 
zu synchronisieren; 

und wobei der Schritt des Sendens eines Pruf- 
punktsignals an jedes der ersten Vielzahl der 
Subsysteme die Schritte der periodischen Er- 
zeugung von Taktsignalen und Prufpunktsigna- 
len durch das ausgewahlte Subsystem an der 
Primarstation und die Ubertragung der Takt- 
und Prufpunktsignale In der Folge ihres Auftre- 
tens an andere Subsysteme, welche die Ko- 
piersatzgruppe an der Primarstation biiden, 
einschlie3lich ihrer selbst, umfaBt; 

und wobei der Schritt des Sendens von Daten- 
aktualisierungen und des Prufpunktsignals an 
jedes der mit einem Gegenstuck verbundenen 
zweiten Vielzahl der Subsysteme fur jedes 
Subsystem in der Kopiersatzgruppe die asyn- 
chrone Blldung einer lokalen Folge von aktua- 
lisierten Datensatzen umfaBt, wobei die Takt- 
und Prufpunktsignale in die Folge eingebunden 
werden, um eine nach dem Zeitpunkt unter- 
schiedene Gesamtanordnung von aktualisier- 
ten Datensatzen zu biiden, und Senden von 



IS 

an der Primarstation als Antwort auf die Einlei- 
tung einer Startkopieroperation an jedem der 
Subsysteme in der ersten Vielzahl der Subsy- 
steme Feststellen der Untergruppe der Vielzahl 
20 der Subsysteme, die eine Kopiersatzgruppe 

bildet, und Bestimmen eines der Vielzahl der 
Subsysteme zu einer Takt- und Prufpunktsi- 
gnalquelle, wobei jedes Prufpunktsignal einen 
Folgetaktwert und eine erhohte Folgenummer 
25 enthalt; 



Aktivieren (202) eines jeden der ersten Vielzahl 
der Subsysteme, damit es mit den anderen 
Subsystemen in der ersten Vielzahl der Subsy- 
steme kommunizlert; 

35 

Aktivieren eines jeden der ersten Vielzahl der 
Subsysteme, damit es mit seinem Subsystem- 
Gegenstuck In der zweiten Vielzahl der Subsy- 
steme, mit dem es verbunden ist, kommunl- 
ziert; 

Erstellen (206) von mindestens einer Konflgu- 
rationstabelle in jedem der ersten Vielzahl der 
Subsysteme, so daB jedes der ersten Vielzahl 
der Subsysteme alle anderen Subsysteme in 45 
der ersten Vielzahl der Subsysteme erkennen 
kann; und 

Synchronisieren (208) der ersten Vielzahl der 

Subsysteme. 50 
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mindestens einem Tell der Folge an das Sub- 
system-Gegenstuck an der Sekundarstation; 

und wobei der Schritt der Koordination des 
Schreibens von Datenaktualisierungen und 
Prufpunktsignalen an der Sekundarstation die 
Schritte des Sendens (410) des Prufpunktsi- 
gnals an das ausgewahlte Subsystem an der 
Sekundarstation durch jedes Subsystem in der 
Kopiersatzgruppe und das Schrelben der Fol- 
gen Oder von Tellen der Folgen, die in den Sub- 
systemen gepuffert sind, erst nach einem Si- 
gnal von dem ausgewahlten Subsystem, das 
dessen Empfang aller an das Subsystem ge- 
sandten Prufpunktsignale (412) anzeigt, um- 
faBt. 

7. Verfahren nach Anspruch 6, wobei jedes PrQfpunkt- 

signal anzeigt, da3 alle Datenaktualisierungen mit 
einem Taktsignal, dessen Folgenummer kleinerals 
der PrufpunktsignalFolgetaktwert ist. an das sekun- 
dare Subsystem-Gegenstuck ubertragen worden 
sind. " 

8. Asynchrones Datenfernkopiersystem (20), das eine 
Primarstation enthalt, die eine erste Vielzahl von 
Speicher-Subsystemen (12) hat, die durch ein er- 
stes Kopplungsmittel (16) miteinander verbunden 
sind, und eine von der Primarstation entfernte Se- 
kundarstation, die eine zweite Vielzahl von Spei- 
cher-Subsystemen (14) hat, die durch ein zweites 
Kopplungsmittel miteinander verbunden sind, wo- 
bei jedes der zweiten Vielzahl der Speicher-Subsy- 
steme unabhangig von den anderen mit einem Ge- 
genstOck der ersten Vielzahl der Speicher-Subsy- 
steme verbunden ist, 

wobei die erste Vielzahl der Subsysteme ein 
Mittel (1 2) enthalt, urn an jedes der ersten Viel- 
zahl der Subsysteme ein Prufpunktsignal zu 
senden; und ein Mittel (12), um Datenaktuali- 
sierungen und das Prufpunktsignal an jedes 
der mit einem Gegenstuck verbundenen zwei- 
ten Vielzahl der Subsysteme zu senden; und 

die zweite Vielzahl der Subsysteme ein Mittel 
(14) enthalt, um die Datenaktualisierungen und 
die Prufpunktsignale zu empfangen und zu puf- 
fern; und ein Mittel (14), um das Schreiben der 
Datenaktualisierungen auf der Grundlage der 
Prufpunktsignale zu koordinieren, so daB die 
Integritat der Aktualisierungsfolgen an der Se- 
kundarstation gewahrt bleibt. 

9. System nach Anspruch 8, wobei das Mittel, das da- 
zu dient, ein Prufpunktsignal an jedes der ersten 
Vielzahl der Subsysteme zu senden, des weiteren 
ein Mittel enthalt, um ein Folgesignal an der Primar- 



station zu verteilen, wobei das Prufpunktsignal In- 
formationen enthalt, die den Wert des Folgesignals 
anzeigen. 

5 10. System nach Anspruch 8 oder 9, in dem das Mittel, 
das dazu dient, Datenaktualisierungen und das 
Prufpunktsignal an jedes der mit einem Gegenstuck 
verbundenen zweiten Vielzahl der Subsysteme zu 
senden, ein Mittel umfaBt, das auf das Mittel zum 

10 Senden eines Prufpunktsignals an jedes der ersten 
Vielzahl der Subsysteme anspricht, um das Pruf- 
punktsignal in eine Datenaktuaiisierungsfolge von 
jedem der ersten Vielzahl der Subsysteme an des- 
sen Subsystem-Gegenstuck einzufugen. 

IS 

Revendlcations 

1. Precede d'exploitation d'un systeme de replication 
20 asynchrone et ^ distance des donn6es (20), com- 

prenant un premier site ayant une premiere pluralite 
de sous-systemes de memoire (12) interconnectes 
par un premier moyen de couplage (16), et un site 
secondaire, distant du premier site, ayant une 

2S deuxieme pluralite de sous-systemes de memoire 
(14) interconnectes par un deuxl6me moyen de 
couplage, chacun des sous-systdmes de la deuxld- 
me plurality de sous-systemes de memoire 6tant 
couple independamment ^ son pendant dans la 

30 premiere pluralite de sous-systemes de memoire, 
ledit procedd comprenant les phases qui consistent 
k: 

dans la premiere pluralite de sous-systemes : 
35 envoyer (302) un signal de point de controle k 

chacun des sous-systemes de la premiere plu- 
ralite de sous-systemes ; 
envoyer (304) des mises k jour de donnees et 
le signal de point de controle k chacun des pen- 
40 dants de la deuxieme pluralite de sous-syste- ) 

mes couples ; et 

dans la deuxieme pluralite de sous-systemes : 
recevoir (408) et enregistrer dans une memoire 

45 tampon les mises k jours des donnees et les 

signaux de point de contrSle ; 
et coordonner recriture (414) des mises k jour 
des donnees en se basant sur les signaux de 
point de contrSle pour que I'integrite de la se- 

so quence de mise k jour soit preservee sur le site 

secondaire. 

2. Precede selon la revendication 1 , oCi la phase d'en- 
voi d'un signal point dede controle k chacun des 

55 sous-systemes de la pluralite de sous-systemes 
comprend une phase qui consiste k repartir un si- 
gnal de sequence sur le premier site, ledit signal de 
point de contrdle comprenant I'lnformation indi- 
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quant ia valeur du signal de sequence. 

3. ProcSdd selon la revendication 2, ou iadite phase 
d'envoi de mises 6 jour de donn6es et d'un signal 

de point de contrdle a chacun des pendants couples s 
de la deuxieme pluralite de sous-syst6mes com- 
prend les phases qui consistent ^ : 

engendrer de mani6re asynchrone une se- 
quence de mises ^ jour de donn6es dans cha- io 
cun des sous-syst6mes de Iadite premiere plu- 
rality de sous-system es ; 

classer (108) chacune des dites sequences 
conformement au signal de sequence reparti et is 
au signal de point de contrdle ; et 

transmettre de mani^re asynchrone les se- 
quences de mises a jour, de chaque systeme 
de la premiere pluralite de sous-syst^mes k 20 
son sous-systeme Equivalent. 

4. Proc6d§ selon I'une quelconque des revendications 
precedentes, comprenant en outre, sur le premier 
site, les phases initiales qui consistent k : 25 

activer (202) chacun des sous-syst^mes de la 
premidre pluralite de sous-syst^mes pour com- 

muniquer avec les autres sous-syst^mes de Ia- 
dite premiere pluralite de sous-systemes ; 30 

activer chacun des sous-systemes de la pre- . 
mi6re plurality de sous-syst^mes pour qu'il 
communique avec son pendant dans la deuxie- 
me pluralite de sous-systfemes couples ; 3S 

construire (206) au moins un tableau de confi- 
guration dans chacun des sous:systemes de la 
premiere plurality de sous-syst^mes pour que 
chaque sous-systemes de la premiere pluralite 
de sous-systemes puisse identifier tous les 
autres sous-systemes de Iadite premiere plura- 
lity de sous-systemes ; et 

synchroniser (208) la premiere pluralite de ^5 
sous-systemes. 

5. Proc^de selon I'une quelconque des revendications 
pr6c6dentes. comprenant en outre, sur le deuxieme 
site, les phases initiales suivantes : 50 

recevoir (402) les messages de copies actifs 
dans la deuxieme plurality de sous-systemes ; 

construire (404) des tables actives de copie 
dans la deuxieme plurality de sous-systemes ; 
et 



synchroniser (406) les opy rations de copie de 
la deuxieme pluralite de sous-systemes ; 

et ou la phase de coordination comprend en 
outre la phase qui consiste k exEcuter le ren- 
dez-vous (41 0) de tous les signaux de controle 
dans la deuxieme plurality de sous-systemes. 

6. Procedy selon la revendication 1 , comprenant en 
outre les phases Initiales suivantes : 

sur ie premier site, en ryponse au lancement 
d'une opyration de copie. verifier chacun des 
sous-systemes de Iadite premiere plurality de 
sous-systemes, le sous-ensemble de la plura- 
lity de sous-systemes formant un groupe de co- 
pie, et designer I'un des yiyments de la plurality 
de sous-systemes comme source des signaux 
de synchronisation et de point de controle, cha- 
que signal de point de contrdle comprenant une 
valeur de synchronisation pour la sequence et 
un numero de sequence plus yievy ; 

sur le site secondaire, en reponse au lance- 
ment d'une operation de copie, verifier chacun 
des sous-systemes de Iadite deuxieme plurali- 
ty de sous-systemes, le sous-ensemble com- 
pose de la plurality de sous-systemes formant 
un groupe de copie, et designer I'un des ele- 
ments de la pluralite de sous-systemes comme 
point de rendez-vous des signaux de point de 
contrdle pour synchroniser la deuxieme plura- 
lity de sous-systemes ; 

et ou la phase d'envoi d'un signal de point de 
contrdle k chaque eiyment de la premiere plu- 
rality de sous-systemes comprend les phases 
qui consistent k faire genyrer pyriodiquement, 
par ledit sous-systeme designe sur le site pri- 
maire, des signaux de synchronisation et des 
signaux de point de contrdle et k diffuser les 
dits signaux de synchronisation et de point de 
contrdle, une fois cryys, vers les autres sous- 
systemes du groupe de copie sur le site primai- 
re, y compris vers lui-meme ; 

et ou la phase d'envoi des mises k jour de don- 
nyes et du signal de point de contrdle k chaque 
equivalent dans la deuxieme plurality de sous- 
systemes couples comprend les etapes sui- 
vantes pour chaque sous-systeme du groupe 
de copie : former de maniere asynchrone une 
sequence locale avec les enregistrements mis 
k jour, imbriquer les dits signaux de synchroni- 
sation et de point de controle dans cette sy- 
quence pour former un classement complet 
distinguant dans le temps les enregistrements 
mis k jour, et envoyer au moins une partie de 
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la sequence au sous-syst^me Equivalent dans 
le site secondaire ; 

at ou la phase qui consiste k coordonner r^crl- 
ture des mises a jour des donn6es et les si- 5 
gnaux de point de controle sur le site secondai- 
re comprend les phases suivantes : chaque 
sous-syst^me du groupe de copie envoie (410) 
le signal de point de controle au sous-systeme 
qui a 6t6 d6sign6 dans le site secondaire, et les io 
sequences ou les parties des sequences enre- 
gistrees dans les memoires tampons des dits 
sous-systemes ne sent 6crltes que lorsque ledit 
sous-syst^me d6sign6 indlque par un signal 
qu'il a regu tous les signaux de point de controle 
qui lui ont ete envoyes (412). 

7. Procede selon la revendication 6, ou chaque signal 
de point de controle indique que toutes les mises k 
jourde donn6esayant un signal de synchronisation 20 
avec un numero de sequence inferieur a la valeur 

de synchronisation de sequence du signal de point 
de contrdle ont 6X6 transmises au sous-syst^me se- 
condaire Equivalent. 

25 

8. Syst6me de r6plication de donnEes, asynchrone, k 
distance (20) incluant un site principal ayant une 
premiere plurality de sous-syst§mes de mEmoire 
(12) interconnectEs par un premier moyen de cou- 
plage (1 6), et un site secondaire distant du site prin- 30 
cipal ayant une deuxiEme plurality de sous-syste- 
mes de mEmoire (14) interconnectEs par un deuxie- 

me moyen de couplage. chacun des systEmes de 
la deuxieme pluralite de sous-systemes etant cou- 
ple independamment k son pendant dans la pre- 35 
mlEre plurality de sous-systEmes de mEmoire, 

la premiere plurality de sous-syst6mes com- 
prenant un moyen (12) pour envoyer un signal 
de point de controle a chaque systeme de la 40 
premiere pluralite de sous-systemes; et un 
moyen (12) pour envoyer les mises k jour des 
donnEes et le signal de point de controle k cha- 
que pendant couple de la deuxiEme plurality de 
sous-systemes ; et 45 

la deuxiEme plurality de sous-systEmes com- 
prenant un moyen (14) pour recevoir et conser- 
ver dans une mEmoire tampon les mises k jour 
des donnees et les signaux de point de ^0 
contrdle; et un moyen (14) pour coordonner 
I'Ecriture des mises k jour de donnEes en se 
basant sur les signaux de point de contrdle de 
telle sorte que I'integritE de la s6quence de mi- 
se k jour soit pr6serv6e sur le site secondaire. 55 

9. SystEme selon la revendication 8, ou le moyen pour 
envoyer un signal de point de contrdle k chaque 



systEme de la premiere plurality de sous-systEmes 
comprend en outre un moyen pour rEpartir un signal 
de sequence sur le site principal, ledit signal de 
point de contrdle comprenant I'information indi- 
quant la valeur du signal de s6quence. 

10. SystEme selon la revendication 8 ou 9. dans lequel 
le moyen pour envoyer les mises k jour de donnEes 
et le signal de point de controle k chacun des sys- 
tEmes pendants de la deuxieme plurality de sous- 
systEmes couplEs comprend un moyen qui rEpond 
au moyen qui envoie le signal de point de contrdle 
k chaque systEme de la premiere plurality de sous- 
systemes en insurant le signal de point de contrdle 
dans la sequence de mise a jour de donnees en- 
voyee par chaque sous-systeme de ladite premiere 
plurality de sous-systdmes k son pendant. 
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